Text-to-text generation models have increasingly become the go-to solution for a wide variety of sequence labeling tasks (e.g., entity extraction and dialog slot filling). While most research has focused on the labeling accuracy, a key aspect -- of vital practical importance -- has slipped through the cracks: understanding model confidence. More specifically, we lack a principled understanding of how to reliably gauge the confidence of a model in its predictions for each labeled span. This paper aims to provide some empirical insights on estimating model confidence for generative sequence labeling. Most notably, we find that simply using the decoder's output probabilities is not the best in realizing well-calibrated confidence estimates. As verified over six public datasets of different tasks, we show that our proposed approach -- which leverages statistics from top-$k$ predictions by a beam search -- significantly reduces calibration errors of the predictions of a generative sequence labeling model.

顺序标记是一项基本的NLP任务，构成了许多应用程序的骨干。对SEQ2SEQ模型的监督学习（如T5）在这些问题上取得了巨大的成功。但是，这些模型的培训目标与我们在实际应用中关心的指标和Desiderata之间存在显着脱节。例如，实用的序列标记应用程序可能需要优化某些Precision-Recall折衷（TOP-K预测），这与最大化金标记序列的可能性的标准目标完全不同。因此，为了弥合这一差距，我们提出了Groot，这是一个简单而有效的框架，用于生成文本序列的奖励优化。 Groot通过训练生成的顺序标记模型来工作，以将解码器输出分布与（Black-Box）奖励函数的输出分布相匹配。使用迭代培训制度，我们首先生成预测候选者，然后纠正其中的错误，最后对比这些候选者（基于其奖励价值）。正如通过四个公共基准测试的广泛实验所证明的那样，Groot显着改善了所有奖励指标。此外，Groot还导致了整体解码器分布的改善，这是由顶级$ K $候选者的质量提高所证明的。

检索演示的生成模型比独立语言模型提供了许多好处：除了对给定查询的文字答案外，它们还提供了从可更新知识库中检索到的出处项目。但是，它们也是更复杂的系统，需要处理长输入。在这项工作中，我们介绍了FID Light，以强烈提高最先进的检索功能模型的效率，同时保持相同的有效性。我们的FID光模型将信息流从编码器（分别编码段落）限制为解码器（使用串联编码表示）。此外，我们通过文本源指针通过重新排列的功能调整FID光，以提高排名最高的出处精度。我们对七个知识密集任务（KILT）的各种实验表明，FID光线始终改善了查询潜伏期和有效性之间的帕累托前沿。带有源指向的FID光设置为六个苏格兰短裙任务的新最新结果，用于合并文本生成和出处检索评估，同时保持合理的效率。

我们引入了一个有效的视频分割系统，用于利用异构计算的资源有限的边缘设备。具体而言，我们通过在已经轻巧的骨架上跨越规格的多个规范来设计网络模型，以市场可用的边缘推理引擎为目标。我们进一步分析和优化了CPU，GPU和NPU的系统中的异质数据流。从经验上讲，我们的方法已经很好地考虑了我们的实时AR系统，通过三倍的有效分辨率使精度更高，但在端到端延迟，较高的帧速率甚至更低的电力消耗下，在Edge平台上也可以使用。

本文研究了针对知识密集型任务的检索型生成模型的多任务培训。我们建议通过利用知识密集型一代的独特属性来清理设定的训练：查询 - 答案对与知识库中的项目的联系。我们通过对相关性标签的信心阈值过滤训练示例，无论一对是通过知识库而回答的。我们在苏格兰语基准的七个组合任务上训练一个单一的数字化（FID）发电机。实验结果表明，我们的简单而有效的方法基本上改善了两个强烈不平衡任务的基线。并显示其余任务的改进较小或没有重大回归。此外，我们通过相关性标签采样量表很好地展示了我们的多任务培训，并且具有增加的模型容量，并实现了最先进的训练，并在七个苏格兰短裙任务中五个。

Quadruped robots are currently used in industrial robotics as mechanical aid to automate several routine tasks. However, presently, the usage of such a robot in a domestic setting is still very much a part of the research. This paper discusses the understanding and virtual simulation of such a robot capable of detecting and understanding human emotions, generating its gait, and responding via sounds and expression on a screen. To this end, we use a combination of reinforcement learning and software engineering concepts to simulate a quadruped robot that can understand emotions, navigate through various terrains and detect sound sources, and respond to emotions using audio-visual feedback. This paper aims to establish the framework of simulating a quadruped robot that is emotionally intelligent and can primarily respond to audio-visual stimuli using motor or audio response. The emotion detection from the speech was not as performant as ERANNs or Zeta Policy learning, still managing an accuracy of 63.5%. The video emotion detection system produced results that are almost at par with the state of the art, with an accuracy of 99.66%. Due to its "on-policy" learning process, the PPO algorithm was extremely rapid to learn, allowing the simulated dog to demonstrate a remarkably seamless gait across the different cadences and variations. This enabled the quadruped robot to respond to generated stimuli, allowing us to conclude that it functions as predicted and satisfies the aim of this work.

We consider the task of text generation in language models with constraints specified in natural language. To this end, we first create a challenging benchmark Cognac that provides as input to the model a topic with example text, along with a constraint on text to be avoided. Unlike prior work, our benchmark contains knowledge-intensive constraints sourced from databases like Wordnet and Wikidata, which allows for straightforward evaluation while striking a balance between broad attribute-level and narrow lexical-level controls. We find that even state-of-the-art language models like GPT-3 fail often on this task, and propose a solution to leverage a language model's own internal knowledge to guide generation. Our method, called CognacGen, first queries the language model to generate guidance terms for a specified topic or constraint, and uses the guidance to modify the model's token generation probabilities. We propose three forms of guidance (binary verifier, top-k tokens, textual example), and employ prefix-tuning approaches to distill the guidance to tackle diverse natural language constraints. Through extensive empirical evaluations, we demonstrate that CognacGen can successfully generalize to unseen instructions and outperform competitive baselines in generating constraint conforming text.

Language models have been shown to perform better with an increase in scale on a wide variety of tasks via the in-context learning paradigm. In this paper, we investigate the hypothesis that the ability of a large language model to in-context learn-perform a task is not uniformly spread across all of its underlying components. Using a 66 billion parameter language model (OPT-66B) across a diverse set of 14 downstream tasks, we find this is indeed the case: $\sim$70% of attention heads and $\sim$20% of feed forward networks can be removed with minimal decline in task performance. We find substantial overlap in the set of attention heads (un)important for in-context learning across tasks and number of in-context examples. We also address our hypothesis through a task-agnostic lens, finding that a small set of attention heads in OPT-66B score highly on their ability to perform primitive induction operations associated with in-context learning, namely, prefix matching and copying. These induction heads overlap with task-specific important heads, suggesting that induction heads are among the heads capable of more sophisticated behaviors associated with in-context learning. Overall, our study provides several insights that indicate large language models may be under-trained to perform in-context learning and opens up questions on how to pre-train language models to more effectively perform in-context learning.

The number of international benchmarking competitions is steadily increasing in various fields of machine learning (ML) research and practice. So far, however, little is known about the common practice as well as bottlenecks faced by the community in tackling the research questions posed. To shed light on the status quo of algorithm development in the specific field of biomedical imaging analysis, we designed an international survey that was issued to all participants of challenges conducted in conjunction with the IEEE ISBI 2021 and MICCAI 2021 conferences (80 competitions in total). The survey covered participants' expertise and working environments, their chosen strategies, as well as algorithm characteristics. A median of 72% challenge participants took part in the survey. According to our results, knowledge exchange was the primary incentive (70%) for participation, while the reception of prize money played only a minor role (16%). While a median of 80 working hours was spent on method development, a large portion of participants stated that they did not have enough time for method development (32%). 25% perceived the infrastructure to be a bottleneck. Overall, 94% of all solutions were deep learning-based. Of these, 84% were based on standard architectures. 43% of the respondents reported that the data samples (e.g., images) were too large to be processed at once. This was most commonly addressed by patch-based training (69%), downsampling (37%), and solving 3D analysis tasks as a series of 2D tasks. K-fold cross-validation on the training set was performed by only 37% of the participants and only 50% of the participants performed ensembling based on multiple identical models (61%) or heterogeneous models (39%). 48% of the respondents applied postprocessing steps.

Fine-tuning pre-trained language models (PLMs) achieves impressive performance on a range of downstream tasks, and their sizes have consequently been getting bigger. Since a different copy of the model is required for each task, this paradigm is infeasible for storage-constrained edge devices like mobile phones. In this paper, we propose SPARTAN, a parameter efficient (PE) and computationally fast architecture for edge devices that adds hierarchically organized sparse memory after each Transformer layer. SPARTAN freezes the PLM parameters and fine-tunes only its memory, thus significantly reducing storage costs by re-using the PLM backbone for different tasks. SPARTAN contains two levels of memory, with only a sparse subset of parents being chosen in the first level for each input, and children cells corresponding to those parents being used to compute an output representation. This sparsity combined with other architecture optimizations improves SPARTAN's throughput by over 90% during inference on a Raspberry Pi 4 when compared to PE baselines (adapters) while also outperforming the latter by 0.1 points on the GLUE benchmark. Further, it can be trained 34% faster in a few-shot setting, while performing within 0.9 points of adapters. Qualitative analysis shows that different parent cells in SPARTAN specialize in different topics, thus dividing responsibility efficiently.